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ABSTRACT 

This  paper  provides  recommended  guidance  for  assessing  the  reliability  and  validity  (R&V)  of  the  tools, 
techniques  and  methods  (TTMs)  developed  to  support  the  planning  and  execution  of  operations  primarily 
intended  to  affect  human  and  social  behaviour.  R&V  are  important  as  they  confirm  the  scientific 
credibility  of  TTMs.  This,  in  turn,  underpins  confidence  in  the  decisions  that  are  facilitated  through  their 
use  by  MoD  staff.  These  guidelines  detail  the  issues  of  R&V  that  are  relevant  to  these  TTMs,  why  they 
should  be  assessed,  and  the  practical  tests  and  procedures  that  should  be  implemented  during  the 
development  phase  and  in  use. 

The  paper  was  originally  published  within  Dstl  as  DSTL/TR18609  vl  in  March  2006.  It  was  written  by 
Stephen  Moore,  working  under  the  team  leadership  of  Philip  Jones.  It  has  been  subject  to  minor  updates 
and  cosmetic  alterations  for  publication  within  NATO  HFM  SY202. 


1.0  INTRODUCTION 

Tools,  techniques  and  methods  that  inform  understanding  of  human  behaviour  are  often  considered  to  be 
more  subjective  than  the  modelling  approaches  commonly  referred  to  as  ‘Hard  Operational  Analysis’, 
which  are  used  to  support  central  MoD  UK  policy,  planning  and  acquisition  decisions.  D  Scrutiny,  MoD 
has  outlined  an  approach  for  increasing  confidence  in  the  reliability  and  validity  of  these  hard  Operational 
Analysis  models,  referred  to  as  Verification  and  Validation  [1], 

The  guidelines  suggested  in  this  report  are  intended  to  complement  the  D  Scrutiny  approach,  but  they 
focus  more  specifically  on  increasing  confidence  in  the  wide  variety  of  tools,  techniques  and  methods  that 
are  used  to  support  the  Human  /  Social  component  of  military  operations.  For  convenience,  the  term 
Influence  Operations  is  used  throughout  the  remainder  of  the  report  as  shorthand  for  this  component. 

This  paper  provides  recommended  guidance  for  assessing  the  reliability  and  validity  of  the  tools, 
techniques  and  methods  developed  to  support  the  planning  and  execution  of  Influence  Operations  intended 
to  affect  human  and  social  behaviour.  Reliability  and  Validity  (R&V)  are  important  because  they  confirm 
the  scientific  credibility  of  tools,  techniques  or  methods  (hereafter  referred  to  as  TTMs).  This,  in  turn, 
underpins  confidence  in  the  decisions  that  are  facilitated  through  their  use  by  MoD  staff.  These  guidelines 
detail  the  issues  of  R&V  that  are  relevant  to  these  TTMs,  why  they  should  be  assessed,  and  the  practical 
tests  and  procedures  that  should  be  implemented  during  the  development  phase  and  in  use. 


2.0  AIM  AND  OBJECTIVES 

The  aim  of  this  paper  is  to  provide  structured  and  pragmatic  guidance  that  will  align  the  assessment  of 
R&V  during  Influence  Ops  TTMs  development  more  closely  with  that  of  traditional  Operational  Analysis 
(OA)  models.  The  key  objectives  are  to  ensure  that  Influence  Ops  TTMs: 
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a)  Are  developed  in  a  sufficiently  robust  fashion  so  that  they  are  rigorous  and  objective. 

b)  Have  been  adequately  validated. 

c)  Have  sufficient  scientific  credibility,  auditability  and  confidence  in  their  reliability  to  justify  their 
practical  use  by  MoD  staff. 

d)  Have  undergone  a  sufficient  assessment  of  any  uncertainty  to  allow  MoD  staff  to  evaluate  the 
benefits  and  risk  of  their  practical  use  or  non-use. 


3.0  STRUCTURE  OF  THESE  GUIDELINES 

This  paper  first  outlines  the  goal  of  R&V  assessment,  which  is  the  evaluation  of  fitness  for  purpose.  It  then 
details  the  recommended  assessment  process,  the  assessment  scale  and  the  process  of  recording  the 
assessment  in  a  log  book.  This  paper  then  provides  detailed  recommendations  on  the  issues  that  should  be 
considered  and  addressed  during  the  assessment  process.  It  concludes  with  a  summary  of  the  key  points 
and  recommendations. 


4.0  SCOPE 

These  guidelines  are  generally  consistent  with  the  overarching  D  Scrutiny  guidance  on  Verification  and 
Validation  (V&V).  However,  they  focus  more  specifically  on  the  R&V  issues  associated  with  the 
subjective  analysis,  interpretation  and  prediction  of  human  and  social  behaviour,  as  well  as  the  greater 
diversity  in  the  purposes  and  methodologies  of  Influence  Ops  TTMs.  This  means  that  certain  aspects  of 
V&V,  such  as  quantitative  systems  testing  or  hypothesis  testing,  may  not  always  be  applicable  to  all 
Influence  Ops  TTMs.  This  is  because  Influence  Ops  TTMs  may  be  partly  or  entirely  qualitative,  and 
because  certain  Influence  Ops  TTMs  may  not  produce  predictions  that  can  be  tested  for  accuracy  against 
specific  real  world  events. 

Influence  Ops  TTMs  can  vary  distinctly  in  their  purpose  and  their  methodology,  they  may  be: 

•  Procedural  tools,  intended  to  guide  MoD  staff  through  a  recommended  approach  to  planning  and 
executing  Influence  Ops. 

•  Workshop  techniques  intended  to  facilitate  discussion  and  debate  between  decision-makers. 

•  Research  methods  intended  to  identify  the  attitudes  and  perceptions  of  a  subject  group. 

•  Analytical  processes  intended  to  provide  predictions  on  the  potential  responses  of  a  target  group 
to  certain  actions. 

These  guidelines  therefore  recommend  a  broad  approach  to  R&V  assessment,  which  accounts  for  the 
diversity  in  these  types  of  TTMs,  and  incorporates  a  number  of  additional  factors  that  are  considered 
pertinent  to  this  type  of  analysis. 

These  guidelines  are  not  intended  to  be  a  procedural  straightjacket  to  the  development  process,  particularly 
to  those  who  are  already  experienced  TTMs  developers.  However,  they  do  constitute  a  ‘good  practice’ 
approach  for  increasing  the  rigour  and  credibility  of  the  analytical  support  to  Influence  Ops  planning  and 
execution.  It  is  therefore  highly  recommended  that  this  approach  is  implemented  as  an  integral  part  of  the 
development  process. 
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4.0  RELIABILITY  AND  VALIDITY  ASSESSMENT 

4.1  Evaluating  Fitness  for  Purpose 

R&V  assessment  should  be  primarily  intended  to  support  an  evaluation  of  whether  Influence  Ops  TTMs 
are  fit  for  purpose.  This  means  that  the  use  of  these  TTMs  should  constitute  a  sound  contribution  to  the 
planning  and  execution  of  Influence  Ops.  R&V  assessment  should  aim  to  establish  whether  TTMs  have 
been  developed,  reviewed  and  tested,  where  appropriate,  to  the  extent  that  there  is  sufficient  confidence 
that  they  can  fulfil  the  intended  purpose  as  reliably  and  validly  as  possible.  It  should  also  establish  whether 
there  is  uncertainty  in  any  part  of  a  TTM,  or  in  any  specific  analytical  outputs,  if  a  TTM  produces  them. 
This  assessment  will  help  an  end  user  to  evaluate  the  benefits  and  risks  of  using,  and  not  using,  a  particular 
TTM.  Furthermore  it  will  help  them  to  balance  the  risk  of  any  uncertainty  against  their  requirement  for 
support  with  a  particular  task. 

Importantly,  TTMs  can  be  evaluated  as  fit  for  one  purpose  but  not  for  another.  TTMs  should  not  be 
categorised  as  valid  or  invalid  in  isolation.  This  is  because  the  overall  evaluation  of  whether  Influence  Ops 
TTMs  are  fit  for  a  particular  purpose  is  ultimately  a  subjective  judgement,  dependent  on  the  specific 
situation  and  the  nature  of  the  end  user’s  purpose  for  the  TTM..  These  guidelines  therefore  also  reject  a 
system  of  accreditation,  as  does  the  D  Scrutiny  guidance,  in  favour  of  a  continuous  review  and  revise 
approach.  Although  the  evaluation  of  fitness  for  purpose  is  a  subjective  judgement,  it  can  be  based  on  the 
confidence  in  the  R&V  of  a  TTM,  which  is  built  up  through  the  R&V  assessment  process.  Fitness  for 
purpose  is  therefore  a  combination  of  both  reliability  and  validity: 

•  Reliability.  The  key  principles  are  consistency  and  repeatability.  Influence  Ops  TTMs  are  reliable 
if  they  can  repeatedly  reproduce  a  process  and  maintain  consistent  accuracy  in  any  outputs.  If  they 
are  reliable  then  they  are  capable  of  fulfilling  their  stated  intention. 

•  Validity.  The  key  principles  are  appropriateness,  accuracy  and  utility.  Influence  Ops  TTMs  are 
valid  if  they  utilise  appropriate  techniques,  theories,  Subject  Matter  Experts  (SMEs),  data  and 
staff,  if  any  analysis  of  a  subject  is  as  accurate  as  possible  to  the  self-interpretation  of  the  subject 
themselves,  and  if  they  are  useful  to  their  intended  end  user.  If  they  are  valid  then  they  are  capable 
of  fulfilling  their  specific  purpose  identified  by  the  end  user. 

4.2  The  Assessment  Process 

The  assessment  of  R&V  should  be  an  iterative  process  that  is  integrated  and  conducted  throughout  the 
development  of  Influence  Ops  TTMs.  It  should  also  then  continue  after  the  TTMs  have  been  developed, 
although  to  a  less  frequent  extent,  to  ensure  the  R&V  has  not  changed  over  time  or  that  the  TTMs  are  fit 
for  a  new  puipose  for  which  they  will  be  utilised.  R&V  assessment  should  therefore  progress  through  a 
review  and  test  phase  detailed  below. 

4.2.1  Review 

Evaluation  by  a  series  of  reviews  of  all  key  R&V  issues,  as  described  in  section  5,  to  include  repeatability, 
consistency,  appropriateness,  accuracy  and  usefulness: 

•  Peer  review  to  provide  a  continual  progress  check  on  development  by  appropriately  qualified 
colleagues. 

•  Expert  review  by  relevant  SMEs  internal  and  external  to  MoD,  both  scientific/academic  and 
military  if  relevant.  This  may  include  scrutiny  by  a  panel  of  external  experts  from  a  variety  of 
backgrounds,  or  exposure  at  a  conference,  symposium  or  other  similar  event. 

•  Customer  review  and  feedback  on  the  utility  of  a  TTM,  the  specific  analytical  outputs  (if  there  are 
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any),  and  the  outcomes  of  its  use. 

•  Continued  expert  review,  post-development,  by  an  Oversight  Group  composed  of  the  TTM 
custodian  and  users,  and  SMEs,  to  ensure  relevancy  and  currency  of  the  TTM. 

4.2.2  Testing  or  Comparison 

•  Test  of  individual  components,  such  as  inter-coder  reliability,  V&V  of  software  tools,  or  comparison 
of  outputs  from  content  analysis  software  with  those  of  manual  analysis. 

•  Assessment  of  the  predictive  capability  (if  this  is  the  function)  in  controlled  test  cases. 

•  Comparison  of  specific  analytical  outputs  /  predictions  (if  there  are  any)  with  similar  analytical 
products  (e.g.  from  similar  TTMs,  possibly  developed  by  another  country). 

•  The  collection  and  analysis  of  data  to  provide  a  comparison  of  outputs  /  predictions  (if  there  are  any) 
with  historical  data  and  actual  developments  in  the  real  world. 

4.3  Assessment  Follow-up  Actions 

The  outcome  of  each  of  these  phases  should  be  followed-up  and  any  amendments  required  to  the  TTMs 
should  be  made  accordingly.  Several  of  these  phases  can  be  conducted  concurrently,  some  can  be 
repeated,  and  others  may  not  actually  be  applicable  to  a  particular  type  of  TTM.  TTMs  developers  should 
aim  to  conduct  as  many  as  possible,  where  feasible  and  appropriate,  and  to  do  this  in  a  systematic  and 
objective  manner.  This  requires  judgement  on  behalf  of  the  developer(s)  and  the  end  user(s)  on  what  will 
be  sufficient  to  evaluate  a  TTM’s  fitness  for  purpose.  Ideally  the  TTM  developer(s)  should  outline  a 
specific  timeline  for  these  phases,  at  the  outset  of  the  development  process,  and  agree  this  with  the  end 
user(s)  before  development  begins.  This  paper  does  not  provide  specific  guidance  on  detailed  testing  plans 
for  particular  TTMs  as  this  is  the  responsibility  and  area  of  expertise  of  the  custodian,  and  the  experts 
involved  in  the  review  phases,  to  determine. 

4.3.1  The  Assessment  Scale 

The  D  Scrutinty  guidelines  recommend  a  three -tier  assessment  scale  as  follows: 

•  Level  0  is  ‘Unvalidated’. 

•  Level  1  is  assessment  ‘by  review’. 

•  Level  2  is  assessment  ‘against  real  events’. 

The  D  Scrutiny  guidelines  explain  that  evidence  may  not  always  be  available  to  cover  the  full  spectrum  of 
assessment  needed,  in  which  case  the  weight  of  assessment  rests  largely  on  the  review  procedures.  These 
guidelines  recommend  a  similar  approach  and  stress  that,  due  to  the  diverse  nature  of  Influence  Ops 
TTMs,  the  ability  to  achieve  these  levels  may  not  be  wholly  applicable  to  all.  The  type  and  purpose  of 
Influence  Ops  TTMs  vary  distinctly,  as  some  produce  specific  analytical  predictions  and/or  situational 
awareness,  whilst  others  facilitate  discussion,  debate  and  planning  processes.  It  is  therefore  recommended 
that  the  R&V  of  each  TTM  is  assessed  to  the  greatest,  practical  extent  possible  that  is  relevant  for  that  type 
of  TTM. 

These  guidelines  recommend  that  for  all  TTMs  that  do  not  produce  specific  analytical  outputs,  such  as 
predictions,  the  aspiration  should  be  to  achieve  Level  1,  where  the  R&V  has  undergone  a  full  assessment 
by  all  the  phases  of  review  listed  in  sub-section  4.2.1.  For  those  TTMs  that  do  involve  some  form  of 
analytical  output  the  aspiration  should  be  to  achieve  both  Level  1  and  Level  2,  where  this  analytical  output 
has  also  been  tested,  assessed  and/or  compared  as  fully  as  possible,  using  the  procedures  listed  in  sub- 
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section  4.2.2. 

Evaluating  Risk.  TTM  developers  and  end  users  should  agree  the  required  level  of  R&V  assessment,  and 
the  risk  they  are  willing  to  accept,  at  the  outset  of  the  development  process.  The  primary  aim  of  R&V 
assessment  is  to  establish  a  sufficient  level  of  confidence  for  the  end  user  to  be  satisfied  that  the  TTMs  are 
fit  for  their  particular  purpose.  The  time  and  effort  invested  in  an  R&V  assessment  should  therefore  not  be 
disproportionate  to  the  value  of  the  increased  confidence  that  can  be  established.  One  of  the  key  objectives 
of  the  R&V  assessment  process  is  to  help  the  end  user  evaluate  the  risk  of  using  and  also  not  using  a  TTM 
to  support  them.  Awareness  of  R&V  issues  will  help  the  end  user  balance  the  risks  associated  with 
uncertainties  in  the  TTM,  and  its  outputs,  against  the  risks  associated  with  the  decisions  the  end  user  is 
making.  For  example,  an  analytical  Influence  Ops  TTM  might  provide  assessments  of  how  subject 
individuals  may  behave  in  certain  situations,  which  may  be  used  to  support  the  planning  of  Influence  Ops 
activities.  However,  if  the  assessment  is  based  on  relatively  little  source  material,  there  may  be  a  lack  of 
confidence  in  the  conclusions.  In  this  case  the  decision-maker  should  be  advised  of  the  risk  of  relying  too 
significantly  on  the  output  and  that  any  planned  Influence  Ops  activities  could  result  in  negative 
unforeseen  or  undesired  consequences. 

4.3.2  The  Log  Book 

It  is  recommended  that  the  R&V  assessment  process  is  recorded  in  a  log  book.  The  production  of  a  log 
book  should  begin  at  the  outset  of  TTM  development  and  it  should  record  all  activity  undertaken  during 
the  R&V  assessment.  A  log  book  brings  together  all  the  information  and  evidence  of  R&V  that  has  been 
conducted  to-date,  including  a  record  of  when  and  where  the  TTM  has  been  used  and,  if  appropriate,  a 
measure  of  its  success.  It  provides  an  audit  trail  for  the  assessment  process  and  highlights  areas  of  R&V 
that  still  need  to  be  improved.  This  records  the  evaluated  fitness  for  a  particular  purpose  and  therefore 
constitutes  a  key  document  that  can  support  an  end  user  in  assessing  the  risks  of  utilising  a  TTM  to 
support  them  with  a  particular  task.  It  also  provides  an  archive  in  which  to  record  valuable  information 
about  a  TTM  to  ensure  that  it  is  not  lost  when  a  particular  individual  is  no  longer  involved  in  using  that 
TTM.  A  custodian  should  be  appointed  for  a  TTM  and  have  responsibility  for  regularly  updating  the  log 
book  to  incorporate  amendments  and  further  developments  to  the  TTM  or  its  R&V  status.  A 
recommended  template  for  this  log  book  has  also  been  produced 

The  log  book  should  include  the  following  sections: 

a)  TTM  name  and  overview  description. 

b)  Intended  domain  of  use. 

c)  TTM  custodian. 

d)  TTM  management  authority  and  any  Intellectual  Property  Rights. 

e)  Relationship  with  other  tools,  techniques  and  methods. 

f)  History  of  development  and  use. 

g)  Requirements:  staff  (training,  skills,  experience),  time,  information,  software,  hardware. 

h)  List  of  available  staff  with  expertise  and  experience. 

i)  Management  control  systems:  boundaries  of  use,  change  control  procedures. 

j)  Accessibility:  ease  of  use. 

k)  Information  management. 

l)  Assumptions,  compromises  and  limitations. 
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m)  TTM  reliability:  repeatability  and  consistency 

n)  TTM  validity:  appropriateness,  accuracy  and  utility. 

o)  Overall  assessment. 

p)  Evaluation  of  fitness  for  purpose. 

q)  Proposals  for  future  assessments  of  reliability  and  validity. 

r)  List  of  references. 

s)  Glossary 

t)  TTM  Reliability  and  Validity  Summary  Sheet  (in  an  Annex). 

5.0  KEY  RELIABILITY  AND  VALIDITY  ISSUES 

The  previous  section  outlined  the  recommended  process  for  conducting  an  R&V  assessment.  This  section 
details  the  key  issues  that  should  be  considered  during  this  process.  These  issues  will  mostly  be  addressed 
during  the  review  phases,  although  some  will  be  relevant  to  the  testing/comparison  phases.  All  issues 
should  be  addressed  as  fully  as  possible,  in  line  with  the  recommendations  in  sub-section  4.3.1.  The 
assessment  results  should  be  recorded  in  the  log  book. 

The  issues  in  this  section  are  considered  particularly  relevant  to  the  R&V  of  Influence  Ops  TTMs  because 
they  involve  the  subjective  analysis,  interpretation  and  prediction  of  human  and  social  behaviour.  These 
activities  are  inherently  subject  to  inconsistency,  bias  and  misunderstanding.  This  section  therefore 
provides  recommended  practical  procedures,  tests  and  approaches  that  can  help  mitigate  these  issues  and 
increase  R&V.  Key  issues  that  will  be  covered  are: 

a)  The  purpose(s)  of  the  TTM. 

b)  The  individual  constituent  parts  of  the  TTM. 

c)  The  underpinning  social  science  theories. 

d)  TheSMEs. 

e)  The  utilisation  of  data. 

f)  The  management  of  staff,  TTM  development  and  TTM  use. 

g)  The  outcomes  (including  specific  analytical  outputs  if  any). 

5.1  The  Purpose(s)  of  a  TTM 

Influence  Ops  TTMs  can  be  developed  for  a  variety  of  different  purposes.  These  may  be  to  include,  but 
not  exclusively,  facilitation  of  Influence  Ops  decision-making  processes,  provision  of  situational 
awareness  on  the  attitudes  of  a  target  audience,  or  provision  of  analysis  to  support  Influence  Ops  activities 
intended  to  influence  specific  groups  or  individuals.  Influence  Ops  TTMs  can  only  be  considered  valid  if 
they  are  actually  capable  of  fulfilling  these  specific  intended  purpose(s).  This  has  several  key  implications: 

a)  There  should  be  an  interested  end-user  community  which  should  have  a  specific  purpose  for  the 
TTM  to  fulfil. 

b)  The  TTM  should  be  capable  of  fulfilling  the  purpose(s)  within  an  agreed  timeframe  and  of  effort. 

c)  The  developer(s)  should  have  a  clear  understanding  of  the  purpose  of  the  TTM. 

d)  Initial  research  should  be  carried  out  to  ensure  that  there  are  no  suitable  existing  alternatives.  If 
new  TTMs  are  to  be  developed  then  they  should  provide  something  that  no  existing  alternative 
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already  does,  or  they  should  fulfil  the  puipose  more  proficiently  and  cost-effectively  than  any 
predecessor. 

An  additional  issue  to  consider  is  that  Influence  Ops  TTMs  will  support  the  planning  and  execution  of 
activities  that  will  affect  people,  who  are  reactive  subjects  that  understand  our  actions  and  can  respond  to 
them.  Influence  Ops  activities  can  therefore  bring  about  unintended  and  undesired  consequences.  Thus,  it 
is  important  that  TTM  developers  advise  and  educate  their  customer  whenever  they  identify  any  key  issues 
that  may  challenge  the  feasibility  of  achieving  the  intended  purpose.  There  are  two  key  steps  that  can 
support  this: 

•  Inter-subjective  analysis.  Where  possible,  analytical  TTMs  should  involve  direct  interaction  and 
consultation  with  the  subject,  which  will  provide  the  most  accurate  interpretation  of  the  attitudes, 
behaviour  and  ethical  norms  informing  that  subject’s  community.  Ideally  this  should  be  conducted 
in  the  language  of  the  subject  or  using  a  translator  if  necessary.  Inter-subjective  analysis  will 
better  identify  whether  the  intended  puipose  is  likely  to  significantly  challenge  current  attitudes, 
behaviour  and  ethical  norms  of  that  subject.  This  informs  the  end  user  of  the  likelihood  of  the 
success  of  an  Influence  Ops  activity.  This  analysis  is  often  used  in  ‘participatory  Information 
Operations’,  which  are  commonly  conducted  in  post-conflict  situations,  where  the  purpose  is  to 
change  the  attitudes  and  behaviour  of  an  indigenous  population. 

•  Indirect  consultation.  Because  Influence  Ops  vary  in  their  type  and  methodology  there  may  be 
cases  where  direct  interaction  with  a  subject  is  not  possible,  such  as  TTMs  that  result  in  the 
production  of  remote  assessments  of  key  individuals.  In  these  cases,  consultation  with  the  closest 
possible  substitute,  such  as  friends,  relatives,  colleagues,  Diaspora  or  SMEs,  is  the  next  best 
alternative  and  should  help  maintain  the  relative  accuracy  of  any  interpretation.  Again,  any  key 
issues  identified  through  this  analysis,  which  may  challenge  the  feasibility  of  the  intended 
purpose,  should  be  relayed  to  the  customer. 

5.2  The  Individual  Constituent  Parts  of  a  TTM 

Influence  Ops  TTMs  may  consist  of  several  individual  constituent  parts.  These  may  be  specific  TTMs, 
such  as  interviews,  facilitated  discussion  sessions,  focus  groups,  questionnaires,  surveys  and/or  content 
analysis  software  tools  (and  this  list  is  not  exhaustive).  These  individual  parts  form  an  element  of  the 
overall  TTM  and  as  such  also  affect  its  overall  R&V.  A  number  of  associated  R&V  issues  should, 
therefore,  be  considered  for  the  following  approaches: 

•  Interviews  and  surveys.  Methods  such  as  surveys  or  interviews  may  not  be  appropriate  for  use  in 
certain  countries,  societies,  cultures,  situations  or  contexts.  If  these  methods  are  to  be  used  then 
the  questions  should  be  carefully  considered  and  selected,  based  on  advice  from  SMEs. 

•  Software  tools.  Where  software  tools  are  used  (e.g.  content  analysis  tools)  to  automate  or  assist  a 
manual  process,  the  comparative  fitness  for  puipose  of  the  approaches  should  be  established.  A 
test  comparison  should  be  conducted.  This  should  highlight  any  notable  differences  between  the 
approaches  and  should  ultimately  support  an  end  user  in  evaluating  the  risks  of  utilising  a 
particular  approach.  For  example,  the  end  user  may  be  willing  to  accept  lower  levels  of  accuracy 
from  automated  coding,  in  exchange  for  the  ability  to  process  large  volumes  of  data  and  a  more 
timely  output. 

•  Focus  groups  and  structured  discussions.  Focus  groups  and  other  facilitated,  structured 
discussion  techniques  should  be  conducted  using  a  consistent  approach  that  is  reproducible  and 
seeks  to  minimise  bias.  Differences  in  the  outputs  should  ideally  be  due  to  differences  in  the 
inputs,  not  in  the  process. 

•  Human  inputs.  The  human  input  to  any  individual  part  of  a  TTM  may  be  susceptible  to 
variability  and  inconsistency.  It  is  necessary  to  ensure  that  different  TTMs  users  are  conducting 
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these  parts  with  a  standardised,  consistent  and  repeatable  approach.  This  may  include  testing 
factors  such  as  inter-coder  reliability  or  ensuring  consistency  of  training  and  skills  in  facilitation 
techniques. 

5.3  The  Underpinning  Theories  of  a  TTM 

Influence  Ops  TTMs  may  utilise  social  science  theories,  from  domains  such  as  psychology,  sociology  or 
international  relations,  to  focus  and  structure  analysis  of  social  behaviour.  However,  these  theories  are 
neither  indisputable  nor  unchanging.  They  are  relative  to  time  and  context,  and  reflect  changes  with  new 
research  discoveries  and  the  consensus  of  opinion  in  the  wider  academic  community.  The  following  issues 
should  therefore  be  considered: 

•  Credibility  of  Theories.  Theories  should  have  sufficient  credibility  within  the  social  science 
domain  from  which  they  are  drawn.  There  should  be  no  widespread  perception  that  a  chosen 
theory  is  highly  controversial,  significantly  flawed  or  can  be  easily  falsified.  There  should  also  be 
significant  supporting  academic  literature  or  research  findings  to  suggest  that  a  chosen  theory  is 
credible  for  supporting  a  particular  type  of  analysis  as  part  of  an  Influence  Ops  TTM.  Any  chosen 
theory  should  be  a  credible  option  amongst  other  alternatives  and  not  selected  solely  due  to  the 
personal  opinion  of  the  TTM  developer.  It  may  be  appropriate  to  take  a  pluralist  approach  and 
combine  more  than  one  theory,  or  even  elements  or  multiple  theories. 

•  Appropriateness  of  Theories.  Theories  should  be  appropriate  for  the  purposes  for  which  they 
will  be  utilised  by  an  Influence  Ops  TTM.  A  chosen  theory  should  ideally  have  been  tested  and 
used  in  previous  research  and,  in  particular,  to  analyse  a  similar  topic  to  that  which  will  be 
analysed  by  the  Influence  Ops  TTM.  Theories  should  also  not  be  narrow  and  biased,  such  as  to 
lead  TTM  users  to  ignore  or  undervalue  relevant  issues,  or  to  bias  any  conclusions  that  are  drawn. 

•  Re-evaluating  Theories.  Theories  should  be  re-evaluated  after  specified  time  periods  to  ensure 
that  they  are  still  credible  and  appropriate.  This  re-evaluation  should  monitor  academic  literature 
and  developments  in  the  social  science  domains,  from  which  theories  are  drawn,  to  identify 
whether  there  is  any  new  research  which  significantly  challenges  the  credibility  or  appropriateness 
of  a  theory.  It  can  also  help  identify  whether  any  new  theories  have  emerged,  which  may  be  more 
credible  and  appropriate  for  the  purpose,  and  therefore  more  valid. 

•  Understanding  and  Incorporating  Theories.  If  social  science  theories  are  to  be  utilised  then 
they  should  be  accurately  interpreted  and  incorporated  into  a  TTM.  A  formal  definition  should  be 
produced  for  any  chosen  theory,  articulating  the  key  principles  and  detailing  how  these  will  be 
applied.  This  can  be  reviewed  by  relevant  SMEs  for  confirmation.  Users  of  TTMs  should  clearly 
understand  this  formal  definition  and  how  the  chosen  theory  is  to  be  applied.  This  can  be 
reinforced  with  standardised  training  to  ensure  a  consistent  level  of  understanding  between 
different  users. 

5.4  Subject  Matter  Experts 

The  process  of  developing  Influence  Ops  TTMs  is  likely  to  involve  the  use  of  SMEs.  SMEs  are  usually 
consulted  for  their  depth  of  knowledge  and  experience  of  a  particular  subject  area.  They  may  be  a  key 
source  of  input  data  and  are  also  often  involved  in  assessing  the  R&V  of  a  TTM.  It  is  therefore  necessary 
to  assess  the  R&V  of  the  SMEs  themselves,  primarily  because  they  are  introducing  an  additional  element 
of  subjectivity  to  the  development  process.  This  assessment  should  focus  on  their  authenticity,  considering 
the  following  factors: 

•  Their  expertise  should  be  based  on  relevant  personal  experience. 

•  They  should  have  a  good  reputation  that  is  well-established  in  their  domain  of  expertise,  such  as 
their  academic  community. 
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•  They  should  be  highly  recommended  by  credible  sources  and  fellow  experts  in  their  domain(s). 

•  They  should  have  a  good  historical  record  of  utilising  their  expertise  to  support  the  development 
or  R&V  assessment  of  these  types  of  TTMs. 

•  The  information  they  provide  should  be  generally  consistent  with  other  sources  (e.g.  empirical 
events  and  SMEs). 

5.5  The  Utilisation  of  Data 

Influence  Ops  TTMs  can  involve  the  use  of  both  quantitative  and  qualitative  data.  This  raises  a  number  of 
R&V  issues  as  the  utilisation  of  data  is  a  subjective  process  and  involves  interpretation.  The  analysis  of 
data  will  always  be  susceptible  to  personal  bias,  misinterpretation  and  cultural  or  social  misunderstanding. 
It  is  therefore  necessary  to  acknowledge  uncertainty  in  both  data  and  the  analysis  process,  and  to  strive  for 
objectivity  in  the  utilisation  of  data  as  far  as  possible. 

•  Data  Sources.  Data  can  be  biased.  If  data  are  inappropriate  or  misleading  then  they  can  invalidate 
an  analytical  output.  The  background  of  input  data  should  always  be  established  and  input  data 
should  ideally  only  be  used  if  they  are  from  a  credible  source.  All  input  data  should,  where 
possible,  have  an  audit  trail.  Validation  of  data  sources  should  include  SMEs,  who  can  function 
as  a  source  of  input  data.  Guidance  should  be  provided  on  preferred  and  recommended  data. 
These  should  be  re-evaluated  on  a  regular,  timely  basis,  to  ensure  continuing  credibility  and 
currency.  A  less  credible  source  of  input  data  may  be  utilised  provided  that: 

•  The  confidence  required  by  the  end  user  permits  some  degree  of  uncertainty. 

•  The  risks  of  utilising  the  data  have  been  evaluated. 

•  The  outputs  dependent  on  it  are  caveated  accordingly. 

•  Data  Interpretation.  Once  data  have  been  obtained  from  credible  sources,  the  nature  of  Influence 
Ops  TTM  means  that  some  interpretation  may  be  necessary  before  they  can  be  utilised.  Where 
possible,  a  standardised  format  or  template  should  be  established  to  guide  TTM  users  in  utilising 
input  data.  The  template  should  detail  the  preferred  approach  for  analysing  and  interpreting  data, 
extracting  information  and  forming  conclusions.  Data  interpretations  and  resulting  conclusions 
from  one  source  should  be  compared  with  another  for  confidence  and  confirmation.  Outputs  from 
TTM  used  to  analyse  the  same  subject  may  be  compared  to  check  that  each  has  interpreted  the 
input  data  and  subjects  similarly.  Where  possible,  interpretations  should  be  reviewed  by  peers  and 
SMEs  to  reduce  the  likelihood  of  personal  prejudice  and  biased  assumptions. 

•  Data  Management.  An  up-to-date  record  should  be  maintained  of  all  data  that  are  used  by  an 
Influence  Ops  TTM.  This  record  should  include  an  assessment  of  all  the  data  sources,  their 
background,  quality,  credibility  and  any  perceived  limits  on  their  accuracy.  It  should  also  provide 
an  audit  trail,  detailing  how  each  item  of  input  data  was  used.  This  will  provide  a  permanent 
testament  to  the  assessments  of  R&V  of  data  that  have  been  conducted. 

5.6  Management  of  TTM  Developers,  Users  and  Use 

The  development  and  use  of  Influence  Ops  TTMs  must  be  managed  appropriately.  The  R&V  of  TTMs 
depends  significantly  on  the  appropriateness  of  the  people  that  develop  and  use  them: 

•  TTMs  developers  should  have  a  sufficient  level  of  skill  and  expertise,  which  may  include 
specialist  knowledge  in  the  subject  area  or  previous  experience  in  TTM  development. 

•  TTM  users  should  have  received  a  sufficient  level  of  relevant  training  to  ensure  that  they 
understand  the  stated  purpose,  how  the  TTM  works,  any  theories  that  are  incorporated,  any 
standardised  approach  for  data  utilisation,  and  how  the  TTM  fulfils  its  purpose.  Standard 
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requirements  should  be  defined  formalising  the  level  of  skill  and  amount  of  training  considered 
necessary  to  use  a  TTM,  including  specific  academic  or  professional  qualifications,  where 
appropriate.  New  and  less-experienced  TTM  users  should  have  a  greater  level  of  review  and 
managerial  oversight  until  they  are  independently  capable  of  operating  to  standards  consistent 
with  other  experienced  users. 

Conditions  of  use.  A  TTM  may  also  only  be  considered  reliable  and  valid  when  used  under  specified 
conditions.  These  conditions  should  be  established  during  the  development  process  and  adhered  to 
consistently.  They  should  include  boundaries  defining  when  a  TTM  should,  and  should  not,  be  used  and 
the  parameters  within  which  it  has  been  evaluated  as  fit  for  purpose.  These  conditions  should  also,  where 
possible,  provide  guidance  on  the  risks  associated  with  using  a  TTM  outside  of  these  boundaries  (e.g.  this 
guidance  should  include  how  to  assess  the  risk,  to  allow  an  evaluation  of  the  limitations  of  the  TTM 
against  the  nature  and  urgency  of  the  decision  that  the  TTM  is  informing). 

5.7  The  Outcomes  of  Using  a  TTM 

Influence  Ops  TTMs  support  MoD  staff  with  planning  and  executing  Influence  Operations.  The  outcomes 
of  using  these  TTMs  vary  depending  on  type  (e.g.  increased  situational  awareness  or  support  for  the 
planning  of  specific  activities).  All  outcomes  should  be  achieved  in  the  context  of  an  ongoing  dialogue 
with  the  end  users.  This  will  keep  them  informed  of  the  R&V  of  a  TTM  and  allow  them  to  evaluate  the 
risks  of  its  utilisation  in  support  of  decision-making.  TTM  fitness  for  the  stated  purpose  is  based  on  two 
key  factors:  process  validity  and  outcome  utility. 

Process  validity.  The  validity  of  an  outcome  partially  depends  on  the  overall  R&V  of  the  process  through 
which  it  was  achieved.  Following  the  guidelines  contained  in  this  document  will  help  ensure  a  TTM  has 
process  validity. 

Outcome  utility.  The  validity  of  the  outcome  also  depends  on  its  utility  to  the  end  user.  Influence  Ops 
TTMs  are  useful  if  they  pragmatically  and  constructively  support  a  decision-maker  with  Influence  Ops 
planning  and  execution.  This  means  they  should  provide  additional  insight  to  a  problem  or  subject  area. 
The  outcome  should  be  achieved  in  an  acceptable  timeframe  and  if  a  specific  analytical  output  is  produced 
it  should  be  delivered  in  a  language  that  is  clearly  understood  by  the  end  user.  In  particular,  using  specific 
social  scientific  terminology  or  jargon  should  be  avoided  or  explained  in  simple  terms.  It  should  be  noted 
that  utility  should  not  be  equated  with  convenience  of  use:  an  unreliable  and  invalid  TTM  should  not  be 
used  to  support  decision-making  just  because  it  is  easy  to  apply.  The  establishment  of  outcome  utility  is 
almost  entirely  dependent  on  receiving  feedback  from  end  users.  Feedback  should  provide  any 
compliments,  constructive  criticisms  and  recommendations,  which  will  allow  the  TTM  to  be  revised 
accordingly  and  help  ensure  it  is  refined  until  it  can  adequately  fulfil  its  purpose. 


6.0  SUMMARY 

A  structured  and  rigorous  approach  to  R&V  assessment  is  required  to  increase  confidence  in  the  TTMs 
used  to  support  the  planning  and  execution  of  Influence  Ops  intended  to  influence  human  and  social 
behaviour. 

This  approach  is  intended  to  complement  existing  MoD  UK  guidance  on  the  V&V  of  Operational 
Analysis  modelling  capabilities. 

All  TTMs  should  be  subject  to  R&V  assessment  by  review.  Those  TTMs  that  produce  specific  analytical 
outputs  should  also  be  subject  to  additional  testing  in  controlled  test  cases  and  by  comparison  with  outputs 
from  other  similar  tools  and  historical  data,  where  feasible  and  appropriate. 
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The  assessment  process  should  address  a  number  of  key  issues.  They  include:  the  intended  puipose  of  the 
TTM;  the  R&V  of  individual  constituent  parts;  the  underpinning  social  science  theories;  the  SMEs;  the 
utilisation  of  data;  the  management  of  TTM  developers,  users  and  use;  and  the  outcomes  of  use. 

Practical  procedures  for  review  and  assessment  are  contained  within  this  document. 

The  results  of  the  assessment  process  should  be  recorded  in  a  log  book.  A  recommended  template  is 
provided  at  Annex  A.  The  log  book  is  a  key  document  enabling  the  end  user  to  evaluate  fitness  for 
purpose  and  the  risk  of  uncertainty  against  the  urgency  of  their  requirement  for  use  of  the  TTM.  The  log 
book  should  be  continually  updated  to  incorporate  any  developments  and  amendments. 
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